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ABSTRACT 



This study examined the use of employer satisfaction surveys 
in the field of teacher education at a state university. Specifically, the 
issues addressed were whether new graduates' ratings of their job performance 
could substitute for direct employer satisfaction ratings, and if ratings of 
individuals by their supervisors yielded similar results to ratings of new 
graduates in general. A total of 66 students who received undergraduate 
degrees or certifications in education during 1996-97 and were working as 
full-time contracted teachers in spring 1998 completed a survey of their 
perceptions of their preparation in 23 competency areas. Additionally, 304 
principals, including 43 principals of the specific students who completed 
surveys, completed a survey regarding their perceptions of new education 
graduates in general. Results indicated that new teachers and their 
principals had very different perceptions of their preparation for teaching 
with significant differences between the groups on seven of eight preparation 
areas. Results suggest that, within the field of education, institutional 
researchers cannot rely upon graduates' opinions of their preparation for 
employment as a substitute for employer surveys. (Contains 8 references.) 
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The Effect Of Method On Assessing Readiness For Employment 
In the Field of Education 

Abstract 

The purpose of this study was to help guide institutional researchers as they tackle surveys of employer 
satisfaction. Specifically, the issues addressed were whether new graduates’ ratings of their job performance could 
substitute for direct employer satisfaction ratings, and if ratings of individuals by their supervisors yielded similar 
results to ratings of new graduates in general. The study was limited to the field of teacher preparation at one large 
state university. Results indicated that new teachers and their principals had very different perceptions of their 
preparation for teaching. However, ratings of new teacher education graduates in general differed very little from 
ratings of specifically named graduates. 




The Effect Of Method On Assessing Readiness For Employment 
In the Field of Education 

As higher education proceeds further into the age of accountability, increasing demands are being made for 
data showing institutional effectiveness. Much of this push comes from outside the institution and includes 
accreditation agencies, state boards, and legislators. One of the more common requests is for employer satisfaction 
data or similar measures that indicate the institution’s graduates are ready for employment. For example, as part of 
an extensive study of performance funding conducted by the Public Higher Education Program at the Rockefeller 
Institute of Government, researchers found that state policy makers and campus representatives shared many of the 
same preferences for performance indicators. Employer satisfaction surveys were one of the measures generally 
considered the most appropriate (Serban, 1998). 

While external constituencies and campus leaders may consider this a straight-forward request and a valid 
measure of institutional effectiveness, those who must gather and interpret the data know that this is not necessarily 
so. Numerous choices must be made and each carries the researcher down a somewhat different path. 

The first choice is whether the institution can simply survey its graduates about their current jobs and how 
well they were prepared for their work or whether they must survey employers instead of or in addition to graduates. 
Many institutions already conduct alumni surveys. Including questions about job preparation is quick and efficient. 
In addition, the argument can be made that who knows better than the graduates themselves about their job 
preparation. 

Many argue, however, that employers themselves should be surveyed (e.g., Banta et al., 1996). One reason 
is that employers may have a perspective on the graduate’s job performance that the graduate lacks. Another reason 
is that legislators simply want their constituents — employers — to know that their opinion is important. 

If the decision is made that employers must be surveyed, how should they be surveyed? Will a survey to 
employers asking in general how well recent graduates were prepared suffice? Or must the longer route be taken of 
asking graduates where they are employed, seeking their permission to contact their employer, and then asking their 
specific supervisor about their job performance? 

Asking about a specific individual rather than a general “average” for all individuals can make a 
difference. Holding positive perceptions about individuals while simultaneously holding negative perceptions in 
general is a well documented psychological phenomena. In other words, people often do not generalize positive 
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perceptions of individual relationships to the larger group to which the individual belongs (e.g., prejudices about a 
group such as African Americans, Mexicans, gays, or welfare recipients may persist despite liking or even admiring 
an individual representative of that group). Another example is public K-12 education where survey respondents 
typically decry the state of public education while simultaneously professing that the education their children are 
receiving is quite adequate. When respondents are asked to rate the nation’s schools they are most likely rate them 
as average or failing. Yet when parents or educators are asked to rate their own schools they most often give them 
high marks (Langdon, 1998). 

In addition, most of us have observed or directly experienced the impact one negative experience can have 
on perceptions of what has otherwise been positive. Though an employer may have hired ten graduates, if one of 
them is unprepared, that employer may conclude that graduates in general are unprepared because of the power of 
that one negative experience. Though the university has graduated 3,000 well-prepared students, if one is found to 
be lacking in basic communication skills, the story of that one graduate is what people will pass along. 

The issue of general or specific ratings is particularly relevant for institutions such as metropolitan 
universities or community colleges who are in partnership with the community, drawing much of their enrollment 
from the local population and in return seeing most of their graduates employed within the community. The college 
or university is often familiar, therefore, to many community members, and, as the old saying goes, sometimes 
“familiarity breeds contempt.” If the institution has an image problem, institutional researchers should probably ask 
themselves how much general ratings are reflective of this perceived image and how much they are a valid measure 
of graduates’ actual performance on the job. 

While employer satisfaction studies are fairly common, and are even mandated in some states, studies that 
offer comparisons between alumni and employer ratings of alumni preparation are still rare (Smith & Wilson, 1992; 
Womble, 1993). Rarer still are studies of systematic cross-validation of ratings (Annis & Rice, 1992; Raymond et 
al., 1993). North Carolina State University Planning and Analysis (1995) has probably conducted the most 
comprehensive comparison of employer and alumni preparation ratings. They found that employers generally rated 
the alumni higher in terms of preparedness than alumni rated themselves. In fact, there was very little agreement 
between employers and alumni on individual ratings of preparedness. 
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Even less literature was found that compared employers’ ratings of specific graduates compared to ratings 
of recent graduates in general. Both approaches are found separately in the literature, but nothing was found that 
compared the two approaches. 

Purpose of the Study 

The purpose of this study was to see (1) how similarly new graduates and their supervisors rate their 
performance, and (2) how closely employers’ ratings agree when asked about specific graduates and when asked 
about new graduates in general. As an indirect way of assessing the impact of one negative experience on ratings in 
general, mean general ratings were compared based on the number of graduates hired to see if hiring more graduates 
was related to lower ratings. In addition, we wanted to see how much response rates varied when employers were 
asked about specific people and when asked about graduates in general. Did the survey generate a higher response 
rate when a specific name was attached to it? 

Methodology 

The study was limited to the field of teacher preparation at one large state university. This subject area was 
chosen specifically because anecdotal information indicated that school system administrators believed the 
university’s graduates were not well prepared to teach. Others, of course, contended this was not the case. 

To address the issue, it was agreed that new graduates would be contacted and, if teaching, asked about 
their perceptions of their preparation. In addition, they would be asked if we could contact their principals to gain 
their perceptions of the graduate’s preparation. A survey was developed with several College of Education faculty 
that covered the 23 competencies for new teachers that the state had produced. The competencies covered the areas 
of instructional planning (4 items), instructional approaches (4 items), learning environments (4 items), evaluation of 
student competency (2 items), application of technologies (2 items), students as learners (3 items), professional 
development (2 items), and parent/colleague/community relations (2 items). Respondents were asked to indicate if 
they were unprepared (1), somewhat prepared (2), largely prepared (3), or fully prepared (4). 

A total of 243 students who received undergraduate degrees or certifications in education during 1996-97 
were contacted in the spring of 1998 and asked what they were currently doing. About 60% returned their surveys 
and 46% (or 66 students) indicated that they were full-time contracted teachers within the first year of completing 
their degree. This group completed ratings on their educational preparation. They also were asked for permission to 




contact their principals and 80% (52) agreed. Of the principals surveyed, 80% responded within the six week period 
that ended the school year. 

An additional group of 492 principals from the areas that hire most of the university’s graduates were 
surveyed in Fall, 1998, regarding their perceptions of new education graduates in general. Surveys were returned 
by 304 or 62%. The group included the 43 principals who formerly responded to the survey about specific 
graduates. For this group, 19 or 44% responded to the second survey. 

All three administrations of the survey showed high internal consistency reliability for the survey as a 
whole. Where only a few items were included in a sub-scale, reliabilities dropped but still generally remained 
acceptable at .70 or above (see Table 1). 



Table 1. Internal Consistency Reliabilities for the three survey administrations 



Survey scale: 


Teachers’ ratings 


Principals’ 


Principals’ 






specific ratings 


general ratings 


Total (23 items) 


.934 


.953 


.951 


Instructional planning (4 items) 


.806 


.904 


.879 


Instructional approaches (4 items) 


.829 


.898 


.834 


Learning environments (4 items) 


.864 


.934 


.892 


Evaluation of student competency (2 items) 


.773 


.814 


.732 


Application of technologies (2 items) 


.936 


.943 


.950 


Students as learners (3 items) 


.859 


.872 


.836 


Professional development (2 items) 


.877 


.802 


.717 


Parent/Colleague/Community Relations (2 items) 


.789 


.697 


.815 



Group comparisons were conducted using analysis of variance and a .05 level of significance. For the 
principals who supplied both specific and general ratings, a paired-groups t-test was employed. The dependent 
variables were total score (sum of ratings on all 23 items) as well as the eight sub-scales. Each was standardized to 
have a mean of 100 and standard deviation of 10 to facilitate comparisons. 
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Graduates’ vs. Employers’ Ratings 

Results indicated that new teachers and their principals had very different perceptions of their preparation 
for teaching. Of the eight preparation areas included in the survey, new teachers and their principals had statistically 
significant differences in seven of them. In addition, summary scores for the two groups differed significantly 
(F=16.06, df=l,105, p=.0001). In every case, principals saw new teachers as more prepared than new teachers saw 
themselves (see Table 2). 

Table 2. Comparisons of Teachers’ and Principals’ Preparation Ratings 



Area of Preparation: 


Teacher Mean 


Principal Mean 


F-ratio 


Pr>F 


Instructional Planning 


97.78 


103.30 


8.39 


.0046 


Instructional Approaches 


98.60 


102.08 


3.18 


.0776 


Learning Environments 


97.31 


104.01 


12.85 


.0005 


Evaluation of Student Competency 


96.96 


104.53 


16.97 


.0001 


Application of Technologies 


96.84 


104.74 


17.51 


.0001 


Students as Learners 


98.18 


102.77 


5.58 


.0200 


Professional Development 


97.94 


103.07 


6.82 


.0104 


Parent/Colleague/Community Relations 


96.10 


106.04 


31.21 


.0001 


Total Rating 


97.03 


104.42 


16.06 


.0001 



These differences were not orderly, i.e., principals who rated their new teachers the highest were not 
matched with new teachers who also rated themselves highly. The Pearson product- moment correlation of the 
teacher’s and the principal’s overall preparation scores was statistically non-significant (r=.03). The same result was 
found for correlations within each area. 

Principals’ Specific vs. General Ratings 

The key question was whether or not the 43 principals who gave ratings of specific education graduates 
would give similar ratings when asked to rate education graduates in general. Calculating difference scores for the 
19 principals who responded a second time and testing for a mean that was significantly different from zero yielded 
no significant differences for the total difference mean or for mean differences in any of the eight areas except one 



(parent/colleague/community relations, t=2.46, df-16, p=.0256). Since the sample size was small, it is difficult to 
know whether the power was simply not great enough to detect true differences between the two sets of ratings or 
whetlier no true differences existed. Means for specific ratings were somewhat higher in most areas, though non- 
significant. 

Only 19 of the 43 responded to the second survey, thereby introducing the possibility that the responding 
group was not representative of the whole. Comparing the means for principals who returned the second set of 
ratings and those who did not on the specific ratings of graduates indicated that no total score mean difference 
existed between the two groups. Two significant differences were found when tests were conducted for the eight 
preparation areas: instructional planning (F=8.26, df=l,42, p=.0064) and instructional approaches (F=4.67, df=l,42, 
p=.0366). In each case, the principals who did not respond to the second survey gave higher ratings than those who 
did respond. The overall evidence is weak, however, for a consistent bias between the respondents and non- 
respondents. 

In addition, a comparison of means between principals who responded to both surveys and those who only 
received and responded to the second survey asking for general ratings showed no significant differences for the 
overall mean or for means in any of the eight sub- areas. In other words, the principals were similar in their 
perceptions, whether they responded to the first survey or not. 

If one negative experience with a specific graduate unduly influences ratings in general, then we might 
expect to find that the more graduates a principal reported that he/she had hired, the lower the ratings since they had 
had more opportunities to hire a graduate who was unprepared. Again, however, the evidence did not confirm this 
hypothesis. Principals who reported hiring only one graduate in the past three years had mean ratings which did not 
differ significantly from principals who hired two or three graduates or from principals who hired four or more 
either overall or in the sub-areas. Ttie one exception was in teachers’ preparation to apply technology (F=3.27, 
df=2,89, p=.0429). In this case, the fewer graduates hired, the higher the mean score. In addition, no relationship 
was found between number of graduates hired and whether the university’s education graduates were rated as less 
prepared, better prepared or similar to graduates from other universities (x^=1.179, df=4, p=.882). 

Given the preponderance of the evidence, we must conclude, therefore, that asking for general ratings of 
graduates should produce about the same results as asking for ratings of specific graduates. The response rate, 
however, was higher when principals were asked to rate specific graduates compared to graduates in general. 
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Conclusions 



These results indicate that within the field of education, institutional researchers cannot rely upon 
graduates’ opinions of their preparation for employment to serve as a substitute for surveys of employers. In this 
case, employers (principals) saw the graduates as more prepared than graduates (teachers) saw themselves. There 
are several possible reasons for this finding. One reason may be the “imposter effect,” so named because of the 
frequent feeling among first-year teachers that they are just pretending to know what they’re doing in running a 
classroom. Another possible reason for the discrepancy was the timing of the survey. It was given at the end of the 
academic year when teachers were most likely to be reviewing all the things they hadn’t accomplished. Also, 
teachers and principals simply may not be communicating about how well the teacher is performing, and this might 
help explain the lack of correlation between teachers’ and principals’ ratings. 

These findings, however, reaffirm the results of a university- wide survey of undergraduate alumni and their 
supervisors conducted by University Planning and Analysis at North Carolina State University (1995). In their 
study, little agreement was found between ratings of employers and alumni. They too found that alumni provided 
lower ratings than employers did and suggested further research to determine the underlying reasons. 

Results of this study indicate that institutional researchers may be on safe ground in asking for ratings of 
“graduates in general” instead of naming specific graduates. However, while this may work in education where 
pinpointing supervisors is relatively easy and the job role of teacher is fairly uniform, similar results may not be 
found for other majors such as business. Response rates may be higher when names are used, assuming employers 
are not asked to rate too many graduates. Before any strong conclusions can be drawn, it is recommended that the 
study be replicated using graduates from other fields and increasing the sample size. 

The purpose of this study was to help guide institutional researchers as they tackle the difficult area of 
employer satisfaction. Though results from education may not generalize to all other majors, education is typically a 
“hot spot” that legislators and others are interested in. And because of continual presses for accountability in public 
education and the documented lack of confidence in the public education system in general, this study provides an 
opportunity to look at one of the more extreme examples of issues in the area of employer satisfaction. 
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